{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6f32d781-84ab-4463-b995-ecede1b96e35",
   "metadata": {},
   "source": [
    "# 反向传播推导"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "02c69702-2d34-4168-8a57-5c3a7ff3a230",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "864cc3ec-991b-4f3b-a627-b134b1664599",
   "metadata": {},
   "source": [
    "## 定义线性方程 \n",
    " - 我的预测的线形方式是 $$y = 2x + 1$$\n",
    " - 为了简化我们只求w，也就是假设b是已知的 $$y' = wx + 1$$\n",
    " - 定义需要预测的线形方程 linreg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a46588d4-384c-4bcd-a092-35639a57ecc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def linreg(x,w):\n",
    "    return w*x + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "896b720c-e5ba-41c5-b657-58c49550bd3d",
   "metadata": {},
   "source": [
    "## 定义损失函数(最小二乘法)\n",
    "\n",
    " $$ L = \\frac{1}{2}(y'-y)^2 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f4b022eb-ac54-4c58-96ce-3ee977ba9c94",
   "metadata": {},
   "outputs": [],
   "source": [
    "def squared_loss(h_hat,y):\n",
    "    return (h_hat-y)**2/2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1076b03-3ee4-431f-aba6-206cdb7f16bd",
   "metadata": {},
   "source": [
    "## 定义系数w\n",
    "\n",
    " - 真实的w是2\n",
    " - 预测的w初始可以是随机的，我们这里定义1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5032330a-aa82-40fc-926d-6ebbbcce1ff1",
   "metadata": {},
   "outputs": [],
   "source": [
    "true_w = torch.tensor(2)\n",
    "w = torch.tensor([1],dtype=float,requires_grad=True) # requires_grad=True 代表自动求导"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "534e8bb0-c024-44b3-9018-b1d48765dcfe",
   "metadata": {},
   "source": [
    "# 训练数据集\n",
    " - 因为目标函数是 $$ y = 2x + 1 $$\n",
    " - 我们的训练数据集就可以从该方程中任选一些 $$ (1,3) , (2,5) , (3,7) $$\n",
    " - 比如这里我们选用 $$ (1,3) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1833bbdb-0d9a-44c6-894e-b2e045897f78",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_sample = torch.tensor(1)\n",
    "y_sample = torch.tensor(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed6c799f-9a27-40e5-8b10-8f9827ba9dca",
   "metadata": {},
   "source": [
    "## 用深度学习中术语重新整理赋值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4d1e94ce-8478-474d-b706-5bbc5f7e5ad1",
   "metadata": {},
   "outputs": [],
   "source": [
    "net = linreg\n",
    "loss = squared_loss\n",
    "learn_rate = 0.1 #学习率"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aa64d59-fb69-432c-a3a4-72844c827b2d",
   "metadata": {},
   "source": [
    "## 第一次“训练”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c0280c6-6a67-482a-aba4-6e3f9967ab93",
   "metadata": {},
   "source": [
    " - w为1时的预测值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e00115de-e234-455d-a209-9fa846f5bf67",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2.], dtype=torch.float64, grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h_hat = net(x_sample,w)\n",
    "h_hat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e08c2f68-f99b-49a5-8a53-f59f325a92c3",
   "metadata": {},
   "source": [
    " - 获取loss值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7d58993e-c54c-4be8-9eac-027df6d99539",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.5000], dtype=torch.float64, grad_fn=<DivBackward0>)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l = loss(h_hat,y_sample)\n",
    "l"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2dcb630-5164-4ea4-8225-1660df1c4208",
   "metadata": {},
   "source": [
    " - 通过反向传播获取损失函数对w的导数 \n",
    "  \n",
    "   $$ \\frac{dl}{dw}=\\frac{dl}{dy'} * \\frac{dy'}{dw} $$\n",
    "\n",
    "   - loss函数对y‘的导数\n",
    "     \n",
    "     $$\\frac{dl}{dy'} = y’ - y $$\n",
    "     $$\\frac{dl}{dy'} = 2 - 3 = -1 $$\n",
    "\n",
    "   - y‘对w的导数\n",
    "  \n",
    "     $$\\frac{dy'}{dw} = x = 1 $$\n",
    "\n",
    "   - 通过的链式传导的原理\n",
    "  \n",
    "     $$ \\frac{dl}{dw}=\\frac{dl}{dy'} * \\frac{dy'}{dw} $$\n",
    "     $$ \\frac{dl}{dw}=-1 * 1 = -1 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8faf64bf-5e5d-4898-9c8b-640bf59f168c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-1.], dtype=torch.float64)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l.backward()\n",
    "w.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "142fde9c-686a-49d3-b1fd-5597fd194aeb",
   "metadata": {},
   "source": [
    " - 更新权重w的值\n",
    "\n",
    "   $$ w_{k+1} = w_k - lr*\\frac{dl}{dw}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "dcdd16d6-1b56-4af0-a54e-38270455eac8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.1000], dtype=torch.float64, requires_grad=True)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    w -= learn_rate*w.grad\n",
    "    w.grad.zero_()\n",
    "w"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4aedd9f-bd30-499c-8cc7-56cf1c4c4164",
   "metadata": {},
   "source": [
    "## 第二次“训练”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26db9c95-e64d-4ec2-b456-dc49c1b283cf",
   "metadata": {},
   "source": [
    " - w为刚训练后的1.1\n",
    " - 训练数据集 $$ (1,3) , (2,5) , (3,7) $$\n",
    " - 比如这里我们选用 $$ (2,5) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b2d60566-8b3c-469e-a5b0-eb929d0bf1ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.1000], dtype=torch.float64, requires_grad=True)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_sample = torch.tensor(2)\n",
    "y_sample = torch.tensor(5)\n",
    "w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c0fd7d13-3893-4940-bb35-d620beb1d8b9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3.2000], dtype=torch.float64, grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h_hat = net(x_sample,w)\n",
    "h_hat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5a14f45-232f-4816-b3ff-d2b34d892a73",
   "metadata": {},
   "source": [
    " - 获取loss值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "38c01b4b-a410-43b2-93b0-f708d4abeb00",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.6200], dtype=torch.float64, grad_fn=<DivBackward0>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l = loss(net(x_sample,w),y_sample)\n",
    "l"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "381c2acb-44d5-4d3b-8a01-95a7f3fabbb5",
   "metadata": {},
   "source": [
    " - 通过反向传播获取损失函数对w的导数 \n",
    "  \n",
    "   $$ \\frac{dl}{dw}=\\frac{dl}{dy'} * \\frac{dy'}{dw} $$\n",
    "\n",
    "   - loss函数对y‘的导数\n",
    "     \n",
    "     $$\\frac{dl}{dy'} = y’ - y $$\n",
    "     $$\\frac{dl}{dy'} = 3.2 - 5 = -1.8 $$\n",
    "\n",
    "   - y‘对w的导数\n",
    "  \n",
    "     $$\\frac{dy'}{dw} = x = 2 $$\n",
    "\n",
    "   - 通过的链式传导的原理\n",
    "  \n",
    "     $$ \\frac{dl}{dw}=\\frac{dl}{dy'} * \\frac{dy'}{dw} $$\n",
    "     $$ \\frac{dl}{dw}=-1.8 * 2 = -3.6 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5d522f64-f371-4bcc-8919-f2db1926b71a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-3.6000], dtype=torch.float64)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l.backward()\n",
    "w.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9a2d4d2-0f58-408a-b7c3-d6910943b841",
   "metadata": {},
   "source": [
    " - 更新权重w的值\n",
    "\n",
    "   $$ w_{k+1} = w_k - lr*\\frac{dl}{dw}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2a5c3c34-1a32-4180-8c0a-44273a1e0236",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w.grad tensor([-3.6000], dtype=torch.float64)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([1.4600], dtype=torch.float64, requires_grad=True)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    print(\"w.grad\",w.grad)\n",
    "    w -= learn_rate*w.grad\n",
    "    w.grad.zero_()\n",
    "w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58b40f0c-1748-490f-8293-b684016dd6de",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
